Learning from Examples and Generalization

We analyze t he issue of generalization in systems that learn from examples as a problem of representation of functions in finite fields. It is shown th at it is not possible to design algorit hms wit h uniformly good generalization pr opert ies in the space of all functions. Th erefore th e pr oblem of achieving good generalization properties becomes meaningful only if the functions being st udied belong to restricted classes. We then prop ose th e implementation of systems endowed with several distin ct (biased) strategies, allowing th e exploration and identification of the functional classes of learning prob lems. Two such st rat e-gies (polynomial learning and weighed majority rule) are developed and tested on t he problems of even-odd and two-or-more cluste rs. 1. Int r o duction Whenever t he algorit hm required to pro gram a spec ific t ask is not kn own or t he t ask itself involves un certain fea t ur es, lea rn ing from examples seems to b e a sensible approach. In cont rast to t he const ruct ion of approxima te represent ation s, which rem ain for ever com mit ted to a set of simplifying as-sumpt ions, learning from exam ples allows for system a t ic improvem ent. It suffices to increase t he size of the training set , and the mor e examples are presented the wiser the system be com es. In concep t learning [1], t he gen er aliza ti on capabilit y of a lea rn ing sys te m is judged by its ability to take into account a number of spec ific ob serva tions and then extract and ret ain t heir most imp or tant commo n fea tures. When learning from t he examp les in a training set, t he issu e of gen eraliza t ion t here-fore conce rns t he pro blem of how well t he sys te m reacts to sit uat ions t ha t are not presen ted during t he t raining p er iod. This pro blem has b een addressed by many a u t ho rs [2-13] in t he last few yea rs. The work has been carr ied out mostly in t he context of spe cific archit ect ures-neur al net works or …